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ABSTRACT 

A study compared the test scores of National Fire 
Academy (NFA) students who received on-campus classroom instruction 
and students who received the same instruction delivered by satellite 
teleconference. Following a review of literature that indicated no 
significant difference in test scores between on-campus classroom 
students and teleconference students, a controlled study uas set up. 
The subjects were males between 35 and 55 years of age who held chief 
officer positions in fire departments around the country. The control 
group consisted of 34 students attending an executive development 
course at the NFA, and the experimental group consisted of 35 
students attending a teleconference site. The test instrument 
developed for this study was a criterion-referenced, multiple-choice, 
10-item test, examined for reliability and validity. Both groups 
received identical lectures, onp live and one taped. The satellite 
teleconference group had a mean score of 7.3, whereas the control 
group had a mean score of 9.6. It is concluded that although the 
on-campus group scored higher on the test, both groups did learn. The 
difference may be in student motivation to use the information; 
however, the study demonstrated that teleconferencing was less 
effective than classroom instruction. It is recommended that 
comprehensive instructional developmanr. technology should be included 
in the design and implementation of teleconferences in order to 
increase learning. (KC) 
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The National Fire Academy (NFA) delivers part of its instruction by 
satellite teleconference. -This method of instruction has never been compared 
to on-campus classroom instruction. The problem is that faculty at the NFA do 
not know if one method of delivery is more effective than the other. 

The purpose of this study is to compare the test scores between students 
that receive an on-campus classroom instructional method and students that 
receive the same instruction delivered by satellite teleconference 
instruction. This study is quasi-experimental research. The research 
hypothesis is: There will be no statistically significant difference in test 
scores between NFA students that receive on-campus classroom instruction and 
students that receive the same instruction by satellite teleconference 
instruction. 

The NFA's mission is to increase the professionalism of the nation's 
fire service and other allied professionals engaged in fire prevention and 
control activities. To accomplish its mission, the NFA conducts on-campus 
course? for 4000 students annually, field based courses for 15,0C0 students 
annually, and teleconferences which reach approximately 75,000 students during 
each broadcast. Annually there are three to six teleconferences conducted by 
the NFA faculty. For the first time NFA management is asking if the 
teleconference is an effective instructional methodology, in terms of student 
learning . 
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The literature reviewed for this study included information on 
instructional television, telecourses, videotaped instruction, and live video 
teleconferencing. of the four areas, live video teleconferencing is the 
newest (Barker, 1987:42). There were no studies found that compared the 
effectiveness of classroom instruction to satellite teleconference 
instruction. Brown (1988:24) states that, "Earlier research on development 
and application for educational television may be applicable to the vj.deo 
teleconference . " 

The specific area of interest to this study is the comparison of 
classroom instruction to instruction using television. "Many studies, 
including those at the college and university level, have shown that students 
who learn through telecourses achieve at least as well as those who receive 
traditional instruction" (Daniel, 1988:14). When Dutton (1988) compared the 
test results of graduate engineering students who received classroom 
instruction through videotape, of the classroom instruction, the results 
indicated no statistical significant difference. 

The conclusions of Daniel and Dutton (1988) are not unexpected because 
there have been hundreds of studies comparing the effectiveness of classroom 
teaching to instructional television. Chu and Schramm i eviewed the literature 
comparing classroom instruction to instructional T.V. from 1957 through 1966 
and concluded, "...all these summaries show that in the great majority of 
comparative studies, there is no significant difference between learning from 
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television and learning from conventional teaching; and that where there is a 
significant difference, it is a bit more likely to be in favor of television 
than of conventional instruction" (1967:6). A more recent review, of more 
than 100 research reports from 1971 to 1986, by Whittington C987) affirms the 
conclusion that instructional television is as effective as traditional 
methods - 

The literature that was reviewed affirms the research hypothesis of this 
study, that there will be no significant difference in test scores between the 
on-campus classroom student and the teleconference student. This study 
contradicts the literature. 

PROCEDURES 
Population 

The subjects involved in this study were males between 35 and 55 years 
of age who hold chief officer positions in fire departments around the 
country. Subjects were students attending a NFA on-campus course titled 
Executive Development III and students attending a NFA teleconference titled 
Increasing Fire Department Effectiveness Through Organizational Development. 
Both groups of students received the same one-hour module of instruction. 

The control group consisted of 34 students attending an Executive 
Development III course at the NFA in Emmitsburg, Maryland. The Assistant 
Superintendent: designated the class assigned as the control group. Students 
voluntarily applied for the course and were assigned by the NFA's admissions 
office. The only admission requirement is that the student meet the Executive 
Development III student selection criteria as outlined in the course catalog. 
Students must be chief fire officers, directors of state fire agencies, or 
government officials (FEMA, 1987) . 

The experimental group consisted of 35 students attending a 
teleconference site. Teleconference sites are typically fire department 
training centers that have equipment to receive satellite broadcasts. Four 
teleconference sites across the country were designated by the Assistant 
Superintendent as the experimental group. The teleconference program was 
advertised nationally and students voluntarily attend the teleconference site. 
There is no selection criteria for students attending these programs. The 
advertisement for this program states: "This program will be of interest to 
fire service executives, city and county managers, faculty and students and 
allied professionals in public administration" (FEMA, 1988:3). Only two of 
the students in the experimental group met the selection criteria for 
Executive Development in. However, the two groups were equal based cn a 
comparison of there pretest scores. 

Instrumentation 

The test instrument developed specifically for this study was a criteria 
referenced, multiple choice, ten item cest. Content validity was determined 
by three experts. ^ They were an organizational development consultant, an 
evaluation specialist, and a curriculum development specialist. Each person 
compared the test instrument to the lecture outline. The conclusion was that 
the test had item validity because the questions matched the lecture content 
and the test had sampling validity because the questions covered the total 
lecture content (Gay, 1981) . 

Reliability was determined by administering the test to 28 Executive 
Development III graduates and applying a split-half reliability procedure to 
the data. The split-half reliability coefficient was .76. The Spearman-Brown 
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formula indicates a total test reliability of .86. The standard error of 
measurement was .37. The test instrument was accepted as valid and reliable 
for this study. 

Collection of Data 

The Organizational Development module of the Executive Development III 
course was taught during the teleconference and classroom instruction. A 60 
minute video tape of the classroom lecture was made. The video tape was 
broadcast live during the NFA teleconference titled Increasing Fire Department 
Effectiveness Through Organizational Development. On-campus classroom 
students and teleconference students received identical lecture outline 
materials, which were used during the instruction. 

On-campus students were pretested with the instrument developed for this 
study. The test was administered and collected by a NFA faculty member. 
A 5 ter . the P retests were collected, the lecture outline material was 
distributed to the students. A NFA faculty member then conducted the one hour 
lecture. At the conclusion of the lecture a closed book test was 
administered. The posttest was the same instrument used for the pretest and 
it was administered by the same NFA faculty member. 

Teleconference students were pretested with the instrument developed for 
this study. The test was administered and collected by a facilitator at each 
of the four teleconference locations. After the pretests were collected, the 
lecture outline material was distributed; students then viewed the 
teleconference broadcast. At the end of the one-hour lecture, teleconference 
students had the opportunity to phone into the studio to ask questions and 
receive answers about the lecture. The question and answer period lasted 30 
minutes. At the conclusion of the question and answer period, the facilitator 
administered the closed book posttest. The facilitator mailed the pre and 
posttest instruments to the NFA. 

This study was a quasi-experimental design because the control and 
experimental groups could not be randomly selected or assigned. Permission 
from the NFA was granted to use one Executive Development III class and four 
teleconference sites . 

Statistical Analysis 

An independent t test was conducted on the experimental and control 
group pretest scores. ' This was a one-tailed test at a .01 level of 
significance. An independent t test was conducted on the posttest scores. 
This was also a one-tailed test at a .01 level of significance. The reason 
for choosing this statistical analysis is as follows. First, the homogeneity 
of the control and experimental groups needed to be determined. By conducting 
a t test on the p/etest, this was determined. Second, the t test was chosen 
because of the small number of cases. Third, the .01 level of significance 
was selected to reduce the chance of committing a type I error. 

Null hypotheses: There will be no statistically significant difference 
between test scores of students that receive on-campus classroom instruction 
(control) and students who receive the same instruction by satellite 
teleconference instruction (experimental) . 

Alternative hypothesis A: The control group will have a significantly 
higher mean test score than the experimental group. 

Alternative hypothesis B: The experimental group will have a 
significantly higher mean test score than the control group. 
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Assumptions and Limitar ions 

The first assumption is that the test instruments were administered to 
the experimental groups correctly by four different facilitators. Detailed 
written and verbal instructions were given to each facilitator- It is assumed 
that the facilitators followed the instruction exactly. The second assumption 
is that the control and experimental groups are homogeneous in terms of 
intelligence and motivation. 

There are four limitations that threaten the external validity of this 
study. First, pretesting the students sensitized them to the content of the 
lecture. In addition, pretesting is not normally conducted for on-campus 
courses or satellite teleconference instruction. In this study, pretesting 
was conducted in order to help determine internal validity. Second, 
generalization of the findings are questionable because of the small samples 
that were available to this study; an on-campus control group of 3 4 compared 
to a population of 4, 000; a teleconference site experimental group of 35 
compared to a population of 75,000. Third, another problem of generalization 
is caused by the subject matter being studied- The resulcs relate to only one 
module of instruction when there are hundreds of modules of different content 
taught at the NFA. Finally, the lecture, method of instruction is the only 
methodology being used in this study. There are many other methods used at 
the NFA .for both on-campus and teleconference delivery. 

The limitation that threatens the internal validity of the study is the 
differential selection of subjects. Students that volunteer to attend an on- 
campus course may be more knowledgeable about the information being taught 
than the students volunteering to attend the teleconference. Because of this, 
the pretest scores of the control and experimental group were compared. 
Another limitation is the one-hour length of instruction. This was necessary 
to insure that both groups received the exact same instruction. 

Definition of Terms 

The National Fire Academy is part of the Federal Emergency Management 
Agency, under the executive branch of the United States government. 

Course - structured complete blocks or modules of instruction resulting 
in the reward of a certificate for successful completion. 

On-Campus - a training activity delivered using funds from resident 
programs line item of the budget. In this study, it also means a two week 
long course. 

Module - a block of instruction which may be delivered independently or 
as part of a training activity. In this study, it refers to a three-hour 
block of instruction; one-hour of lecture, one-hour of student activity, and 
one-hour of discussion. 

Teleconference - an educational program delivered through r»n electronic 
medium such satellite broadcasting where there is no continual direct 
contact between instructors and students. In this study, a teleconference is 
three-hours of broadcasting which contain both live and video tape delivery as 
well as two way audio communications. 

RESULTS 

The experimental group, teleconference, consisted of 35 students, their 
mean score on the pretest was 6.429, with a low score of 1 and a high score of 
9. The standard deviation was 2.2 97, and the standard error of measure was 
.385 . The control group, on-campus, consisted of 34 students, their mean 
sco^e on the pretest was 7.382, with a low score of 4 and a high score of 10. 
The standard deviation was 1.538, and the standard error of measure was .264. 
An independent t test was conducted at the .01 level of significance. The 
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calculated t was 2.032 and the critical t was 2.666. The calculated t is less 
than the critical t (Table 1) . 

Table 1 

Comparison of Pretest Score3 of 
On-Campus and Teleconference Groups 



Score Standard Standard Error 

Students N X* Low / High Deviation of Measure 



On Campus 34 7.382 4 10 1.538 .264 

Teleconference 35 6.429 ' 9 2 .279 .395 

*Uo statistically significant difference at P=.01, calculated t 2.032, 
critical t 2. 666. 

The posttest result? are as follows. The experimental, satellite 
teleconference, group had a mean score of 7.257, with a low score of 3 ju.i a 
high score of 10. The standard deviation is 2.091 and the standard error of 
measure is .352. The control, on-campus, group had a mean score of 9.588, 
with a low score of 7 and a high score of 10. The standard deviation was .701 
and the standard error of measure was .12. An independent t test was 
conducted at the .01 level of significance. The calculated t was 6.17 and the 
critical t was 2.666. The calculated t was greater than the critical t 
(Table 2) . 

Table 2 

Comparison of Posttest Scores of 
On-Campus and Teleconference Groups 



Score Standard Standard Error 

Students N , X* Low / High Deviation of Measure 



On Campus 34 9.588 7 10 .701 .12 

Teleconference 35 7.257 3 10 2 .091 , 35^ 

*Statistically significant difference at P=.01, calculated t 6.17, critical t 
2.666. 

A comparison between pre and posttest scores for each group was 
conducted. A dependent t test was conducted on both sets of scores. The 
video teleconference group had a calculated t of 2. 666. The critical t at the 
.05 level of significance is 2.042. The calculated t was greater than the 
critical t. The on-campus group had a calculated t of 7.261, the critical t 
at the .001 level of significance was 3.646. The calculated t exceeds the 
critical t (Table 3) . 
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Table 3 



Comparison of Pretest to Posttest Scores of 
On-Campus and Teleconference Groups 





Stvdents 


N 


Pretest 
X 


/ Posttest 
X 


Level of 
Significance 


Calculated 
t 


Critical 
t 


On Campus 


34 


7.382 


9.588 


.001 


7.261 


3.646 


Teleconference 


35 


6.429 


7.257 


.05 


2.666 


2.042 



DISCUSSION 



The null hypothesis was rejected, the research hypothesis was also 
rejected; alternative hypothesis (a) was accepted. The on-campus control 
group scored significantly higher on the posttest than the satellite 
teleconference experimental group. The control mean was 9 588 and the 
experimental mean was 7.257, which is statistically significant at the .01 
level. 

Although the studies of Chu (1967), Wittington (1987), and Dutton (1988) 
support the research hypothesis, the studies cited are comparing classroom 
instruction to instructional television. In these comparisons, the target 
audience for each type on instructional method was the same. The students in 
these studies are attending the same course, its only the medium that was 
different. Brown's (1988) notion that instructional television research may 
apply to satellite teleconferencing was not supported by this study. The 
major difference between the two methodologies is thot instructional 
television implies course and teleconference does not imply course. 

There are a number of differences between NFA on-campus courses and NFA 
teleconferences. On-campus courses are specifically designed for a target 
audience, only th^se "ho meet the target .udience selection criteria can 
attend, and only a few have the opportunity to attend because of limited 
space. Teleconferences, although developed for a specific target eudience, 
have no selection process; literally anyone with a satellite dish car. watch a 
teleconference. This difference in the selection process between the two 
groups was taken into consideration. 

The homogeneity of the two groups was determined by comparing their 
pretest scores to each other; there was no statistically significant 
difference between the scores. The students entry level knowledge of the 
subject matter was the same, but the groups were not equal in their 
organizational position. The control group all meet the student selection 
criteria of being top management. Of the 35 experimental students, only two 
were members of top management and would have met the Executive Development 
III student selection criteria. Because the teleconference group was less 
likely to be able to use the information being taught, it may have impacted on 
their motivation to learn the information. In addition, the information 
taught was designed for a specific target audience, Executive Development III, 
the experimental group did not fall into that target population. This may 
also have negatively influenced their ability to learn as much as the control 
group. 

Despite the fact that the research hypothesis was rejected, both the 
experimental and control group did learn. This is evident from a comparison 
for each group's pre to posttest scores. There was a statistically 
significant difference between the pretest mean of 6.429, for the 
teleconference group, and their posttest mean of 7.257 at trie .05 level of 
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significance. This result does support the literature which indicates that 
students can learn through instructional television. It also indicates that 
students can learn from satellite teleconference instruction when on-campus 
classroom instruction is videotaped and broadcast live. The control group, 
on-campus, had a pretest mean of 7.382 and a posttest mean of 3.588; this is 
statistically significant at the .001 level of significance. The difference 
between the gains in pre to post scores may again be influenced by student 
motivation to use the information and instructional design of the material. 

Finally, although the teleconference students did not learn equally as 
well as the on-campus students, they did learn. Teleconf rencing may not be 
as effective as instructional television or classroom instruction? but it can 
be an effective instructional methodology; when the classroom lecture video 
tape method is used with -students that are homogeneous in regard to their 
entry level knowledge of the content. 

IMPLICATIONS AND RECOMMENDATIONS 

It is evident from this study that the participants do learn from 
teleconferencing, but they do not learn as much as st idents receiving the same 
instruction in the classroom. Teleconferencing is not as effective as on- 
campus classroom instruction. Institutions must decide if the objective of 
teleconferencing is to make it equally as effective as classroom instruction. 
To accomplish this, the same instructional development procedures need to be 
applied to teleconferencing, as are used in on-campus course development and 
delivery. 

To increase the educational effectiveness of teleconferences, 
comprei- ensive instructional development technology must be included in the 
design, implementation, and evaluation stages of teleconference instruction. 
The following three recommendations are made. First, the teleconference 
population needs to be surveyed to determine exactly who watches 
teleconferences. Second, match the teleconference audience to the appropriate 
material from on-campus courses. Third, link a series of teleconferences 
together around the same content and design it as a telecourse. This would 
require students meeting student selection criteria, attendance criteria, and 
successful completion criteria. 

Sacellite teleconferencing is an education tool that can help the 
teachers reach large numbers of students over large geographic areas. 
Students do learn when this tool is used. This new tool's effectiveness, as 
with any tool, is totally dependant in the knowledge, skills,, and abilities of 
the user. 
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